Skip to main content

第 11 章：加密安全與權限控管

Authentication 與 Authorization 概念​

  • Authentication：你是誰？
  • Authorization：你可以做什麼？

所有 Kubernetes 叢集都有兩種類型的使用者

  • service accounts managed by Kubernetes, 程式(pod)連接 APIServer
  • normal users. 普通用戶，例如透過 kubectl 連接 API Server

設定 Certificates and kubeconfig files​

  • kubeadm-based cluster will:
    • create self-signed Certificate Authority (in /etc/kubernetes/pki)
      • ca.key private key
      • ca.crt CA Certificate
        • 會被複製到各個 cluster 節點上，讓 node 信任由這個 CA 簽名的證書。（同時也在 kubeconfig 文件裡）
  • Generates Certificates for System Components
  • kubernetes-admin User created

建立 Certificate​

  • Create new Certificate for new user
  • Create a private key with openssl
  • Create a Certificate signing request with openssl
  • Create and submit CertificateSigning Request object
  • Approve the CertificateSigning Request
  • Retrieve the Certificate
$ # create a private key
$ openssl genrsa -out demouser.key 2048# generate CSR
$ # CN(common name) is your username, o(Organization) is the Group
$ openssl req -new -key demouser.key -out demouser.csr -subj "/CN=demouser"
$ # the CertificateSigning Request needs to be base64 encoded
$ cat demouser.csr | base64 | tr -d "\n"`

Create CertificateSigning Request object

apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
    name: demouser
spec:
    groups:
    - system:authenticated
    request: put the base64 encoded csr here
    signerName: kubernetes.io/kube-apiserver-client
    usages:
    - client auth

Approve CertificateSigning Requst object

$ # approe the CSR
$ kubectl certificate approve demouser

# retrieve the certificate from the CSR object, and decode it from base64
$ kubectl get certificatesigningrequests demouser -o jsonpath='{.status.certificate}' | base64 --decode > demouser.crt

$ # check certificate
$ openssl x509 -in demouser.crt -text

設定 kubeconfig files​

  • 位址：/etc/kubernetes/admin.conf
  • 內容
    • Users
    • Clusters
    • Contexts
建立 demo 用戶
kubectl config set-credentials demouser --client-key=demouser.key --client-certificate=demouser.crt --embed-certs=true
查看 user
kubectl config get-users
建立 context
kubectl config get-contexts
CURRENT   NAME                          CLUSTER      AUTHINFO           NAMESPACE
         kubernetes-admin@kubernetes   kubernetes   kubernetes-admin   default
kubectl config set-context demo --user=demouser --cluster=kubernetes
Context "demo" created.
$ kubectl config get-contexts
CURRENT   NAME                          CLUSTER      AUTHINFO           NAMESPACE
          demo                          kubernetes   demouser
         kubernetes-admin@kubernetes   kubernetes   kubernetes-admin   default
kubectl config use-context demo
Switched to context "demo".
CURRENT   NAME                          CLUSTER      AUTHINFO           NAMESPACE
         demo                          kubernetes   demouser
          kubernetes-admin@kubernetes   kubernetes   kubernetes-admin   default
kubectl get nodes
Error from server (Forbidden): nodes is forbidden: User "demouser" cannot list resource "nodes" in API group "" at the cluster scope
kubectl get nodes -v 6
I0711 20:58:04.364228   65356 loader.go:372] Config loaded from file:  /home/vagrant/.kube/config
I0711 20:58:04.383605   65356 round_trippers.go:553] GET https://192.168.56.10:6443/api/v1/nodes?limit=500 403 Forbidden in 14 milliseconds
I0711 20:58:04.384119   65356 helpers.go:222] server response object: [{
"kind": "Status",
"apiVersion": "v1",
"metadata": {},
"status": "Failure",
"message": "nodes is forbidden: User \"demouser\" cannot list resource \"nodes\" in API group \"\" at the cluster scope",
"reason": "Forbidden",
"details": {
    "kind": "nodes"
},
"code": 403
}]
Error from server (Forbidden): nodes is forbidden: User "demouser" cannot list resource "nodes" in API group "" at the cluster scope
清理 context 與 user
kubectl config use-context kubernetes-admin@kubernetes
kubectl config delete-context demo
kubectl config delete-user demouser

Role Based Access Control 以角色為基礎的存取控制​

  • Role / ClusterRole 定義了可以做什麼
  • RoleBinding / ClusterRoleBinding 定義了誰可以做在這個 Role / ClusterRole 定義

Roles 定義權限​

Roles 代表的是可以對 k8s 的 resource 做什麼操作， 並且是分別 namespace 的

kubectl create role demorole --verb=get,list --resource=pods --namespace ns1
kubectl create role demorole --verb= --resource=pods --namespace ns1

ClusterRoles 定義權限​

Cluster 層級的 Resource， Node、PersistentVolumes。 ( 跨 namespace )

kubectl create clusterrole democlusterrole --verb=get,list --resource=nodes

RoleBinding / ClusterRoleBinding 定義誰可以有權限​

  • RoleBinding
    • kubectl create rolebinding demorolebinding --role=demorole --user=demouser --namespace ns1
  • ClusterRoleBinding
    • kubectl create clusterrolebinding democlusterrolebinding --clusterrole=democlusterrole --user=demouser
  • /api (core group)
    • 負責 K8S 集群運作的核心功能，像namespaces、Pods、RCs、Nodes、bindings、Secret、Service 等物件
  • /apis (named group)
    • named group相較於core group更有組織性，像是/apps、/extension、/networking.k8s.io、/storage.k8s.io、authentication.k8s.io
  • Verb包含：
    • list, get, create, delete, update, watch
  • 如何查看有哪些API Group呢?
    • 可以透過curl https://localhost:6443 -k指令
      • 需要加上https需要的參數，在指令後方加上-key、-cert及-cacert參數即可
      • 透過kubectl proxy 命令，在 Control Plane 節點上以 port 8001 啟動 kubectl proxy client。這種方式是直接參考 kubeconfig 檔案中 User 的 credentials 和 certificates
    • 第二種方法直接在Control Plane節點上輸入kubectl proxy命令，再開另一視窗輸入curl http://localhost:8001 -k這樣一來就可以看到有哪些API Group是 avaliable的，若要特別查看哪些是named API Group，可以透過grep篩選
  • curl http://localhost:8001/apis -k | grep -i "name"

使用場景​

  • Role / RoleBinding 一般用於單一 namespace 去定義權限
  • ClusterRole / ClusterRoleBinding 一般用於所有的 namespace

Role and RoleBinding 測試​

以管理員身分建立一些資源
kubectl config use-context kubernetes-admin@kubernetes
kubectl create namespace ns1
kubectl create deployment web1 --namespace=ns1 --image=gcr.io/google-samples/hello-app:1.0 --port=8080 --replicas=2# test
kubectl auth can-i list pod #yes
kubectl auth can-i list pod --as demouser #no
建立 role and role binding
kubectl create role demorole --verb=get,list --resource=pods --namespace ns1
kubectl create rolebinding demorolebinding --role=demorole --user=demouser --namespace n
測試權限
kubectl auth can-i list pod --as demouser #no
kubectl auth can-i list pod --as demouser --namespace ns1 #yes
kubectl get pods --namespace ns1 --as demouser
kubectl auth can-i delete pod --as demouser --namespace ns1 #no
kubectl auth can-i list node --as demouser --namespace ns1 #no
kubectl auth can-i list deployment --as demouser --namespace ns1 #no
建立 ClusterRole and ClusterRoleBinding 
$ kubectl create clusterrole democlusterrole --verb=list --resource=node
clusterrole.rbac.authorization.k8s.io/democlusterrole created
$ kubectl create clusterrolebinding democlusteerrolebinding --clusterrole=democlusterrole --user=demouser
clusterrolebinding.rbac.authorization.k8s.io/democlusteerrolebinding created
$ kubectl auth can-i list node #yes
$ kubectl auth can-i list node --as demouser yes

Service Account 定義程式用權限​

建立 ServiceAccount​

kubectl create serviceaccount demosa
kubectl describe serviceaccounts demosa

建立包含 ServiceAccount 的 Pod​

serviceacc.yaml
apiVersion: v1
kind: Pod
metadata:  
	name: client
spec:   
	serviceAccount: demosa
   containers:   
	- name: client
     image: xiaopeng163/net-box:latest
     command:      
			- sh
      - -c
      - "sleep 1000000"

查看 service account 的 token

kubectl describe pod client
透過 jsonpath 過濾
kubectl get pods client -o jsonpath='{.spec.containers[0].volumeMounts}' | python3 -m json.tool
[
    {
        "mountPath": "/var/run/secrets/kubernetes.io/serviceaccount",
        "name": "kube-api-access-tvr98",
        "readOnly": true
    }
]
$ kubectl exec -it client -- sh
/omd #
/omd #
/omd # cd /var/run/secrets/kubernetes.io/serviceaccount
/run/secrets/kubernetes.io/serviceaccount # ls -l
total 0
lrwxrwxrwx    1 root     root            13 Jul 16 14:15 ca.crt -> ..data/ca.crt
lrwxrwxrwx    1 root     root            16 Jul 16 14:15 namespace -> ..data/namespace
lrwxrwxrwx    1 root     root            12 Jul 16 14:15 token -> ..data/token
/run/secrets/kubernetes.io/serviceaccount #

ServiceAccount Authentication​

驗證權限
kubectl exec -it client -- sh
/omd # cd /var/run/secrets/kubernetes.io/serviceaccount
/run/secrets/kubernetes.io/serviceaccount # TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
/run/secrets/kubernetes.io/serviceaccount # CACERT=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
/run/secrets/kubernetes.io/serviceaccount #
/run/secrets/kubernetes.io/serviceaccount # curl --cacert $CACERT -X GET https://kubernetes.default.svc.cluster.local/api{  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "forbidden: User \"system:anonymous\" cannot get path \"/api\"",
  "reason": "Forbidden",
  "details": {},
  "code": 403}/run/secrets/kubernetes.io/serviceaccount #
/run/secrets/kubernetes.io/serviceaccount # curl --cacert $CACERT --header "Authorization: Bearer $TOKEN" -X GET https://kubernetes.default.svc.cluster.local/api{  "kind": "APIVersions",
  "versions": [    "v1"
  ],
  "serverAddressByClientCIDRs": [    {      "clientCIDR": "0.0.0.0/0",
      "serverAddress": "192.168.56.10:6443"
    }
驗證訪問集群資源的權限
/run/secrets/kubernetes.io/serviceaccount # curl --cacert $CACERT --header "Authorization: Bearer $TOKEN" -X GET https://kubernetes.default.svc.cluster.local/api/v1/namespaces/default/pods?limit=500{  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "pods is forbidden: User \"system:serviceaccount:default:demosa\" cannot list resource \"pods\" in API group \"\" in the namespace \"default\"",
  "reason": "Forbidden",
  "details": {    "kind": "pods"
  },
  "code": 403}/run/secrets/kubernetes.io/serviceaccount #

ServiceAccount Authorization​

kubectl auth can-i list pods --as=system:serviceaccount:default:demos
I0716 14:46:05.735051   61770 loader.go:372] Config loaded from file:  /home/vagrant/.kube/config
I0716 14:46:05.761522   61770 round_trippers.go:553] GET https://192.168.56.10:6443/api/v1/namespaces/default/pods?limit=500 403 Forbidden in 20 milliseconds
I0716 14:46:05.762209   61770 helpers.go:222] server response object: [{  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "pods is forbidden: User \"system:serviceaccount:default:demosa\" cannot list resource \"pods\" in API group \"\" in the namespace \"default\"",
  "reason": "Forbidden",
  "details": {    "kind": "pods"
  },
  "code": 403}]
Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:default:demosa" cannot list resource "pods" in API group "" in the namespace "default"

RBAC​

建立 RBAC
$ kubectl create role demorole --verb=get,list --resource=pods
$ kubectl create rolebinding demorolebinding --role=demorole --serviceaccount=default:demosa
rolebinding.rbac.authorization.k8s.io/demorolebinding created
$ kubectl auth can-i list pods --as=system:serviceaccount:default:demosa
yes

進入到一個綁定此 service account 的 pod 進行測試

kubectl get nodes
$ kubectl exec -it client -- sh
/omd # cd /var/run/secrets/kubernetes.io/serviceaccount
/run/secrets/kubernetes.io/serviceaccount # TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
/run/secrets/kubernetes.io/serviceaccount # CACERT=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
/run/secrets/kubernetes.io/serviceaccount #
/run/secrets/kubernetes.io/serviceaccount # curl --cacert $CACERT --header "Authorization: Bearer $TOKEN" -X GET https://kubernetes.default.svc.cluster.local/api/v1/namespaces/default/pods?limit=500{  "kind": "PodList",
  "apiVersion": "v1",
  "metadata": {    "resourceVersion": "1625465"
  },
  ....
  ....

Security Context​

它定義了Pod或容器的特權和訪問控制設置。SecurityContext包括：

  • Discretionary Access Control

    訪問目標（如檔案）的權限基於User ID（UID）和 Group ID（GID）。

  • Security Enhanced Linux (SELinux)

    為目標分配安全標籤

  • Running as privileged or unprivileged

    以特權或非特權運行

  • Linux Capabilities

    為某些 process 提供特權，但不是 root 的所有特權

  • AppArmor

    使用程式配置文件來限制個別程式的功能。

  • Seccomp

    過濾及篩選 process 的 system call

  • AllowPrivilegeEscalation

    控制 process 是否可以比其 parent process 獲得更多的特權

  • readOnlyRootFilesystem

    將容器的 root file system mount 為 Read-Only

在 Kubernetes 內也可以透過 SecurityContext 對每個 Container 去進行相關的權限設定。要為Pod指定安全設置，需在Pod YAML中加入securityContext參數

cat security-context.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: security-context-demo
spec:
  securityContext:
    runAsUser: 1000
    runAsGroup: 3000
    fsGroup: 2000
  volumes:
  - name: sec-ctx-vol
    emptyDir: {}
  containers:
  - name: sec-ctx-demo
    image: busybox
    command: [ "sh", "-c", "sleep 1h" ]
    volumeMounts:
    - name: sec-ctx-vol
      mountPath: /data/demo
    securityContext:
      allowPrivilegeEscalation: false
  • runAsUser: 1000 對於 Pod 中的任何容器，所有 process 都已 User ID = 1000 運行
  • runAsGroup: 3000 Pod 的任何容器內的所有 process 指定 primary group ID 為 3000。指定runAsGroup 時， User ID 1000 和 Group ID 3000 也將擁有所有創建的檔案。
    • K8s default primary group ID: root(0))
  • fsGroup: 2000 容器的所有 process 也是 supplementary Group ID 2000 的一部分。Volume /data/demo 及該 Vloume 中所有檔案的擁有者均為 Group ID 2000。